feat(nomad): add metadata_options to enable imdsv2 #297

Tingweiftw · 2023-11-24T08:00:15Z

Motivation and Context

https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/

Amazon recommends the migration to IMDSv2, a session based approach to accessing metadata to the instances.

Description

As hashicorp has archived their terraform-aws-vault, terraform-aws-consul, and terraform-aws-nomad public repository, we cannot contribute to the open source and edit the downstream modules to enable IMDSv2 in the launch configurations for these service. The solution is to copy the source code pinned to the version and edit from there, as seen from the example of nomad-cluster.

Copied folders for consul

consul-cluster
consul-iam-policies
consul-security-group-rules
consul-client-security-group-rules
Copied folders for vault
vault-cluster
vault-security-group-rules

How has this been tested?

For nomad servers (don't have the diff for nomad client, but similar)

Terraform will perform the following actions:

  # module.core.module.nomad_servers.aws_autoscaling_group.autoscaling_group will be updated in-place
  ~ resource "aws_autoscaling_group" "autoscaling_group" {
      + force_delete_warm_pool    = false
        id                        = "vm-govtech-locus-stgez-nomads"
      ~ launch_configuration      = "vm-govtech-locus-stgez-nomads-20230905041931161400000001" -> (known after apply)
        name                      = "vm-govtech-locus-stgez-nomads"
        # (21 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

  # module.core.module.nomad_servers.aws_launch_configuration.launch_configuration must be replaced
+/- resource "aws_launch_configuration" "launch_configuration" {
      ~ arn                              = "arn:aws:autoscaling:ap-southeast-1:802909931189:launchConfiguration:b40ff87c-5596-445d-9744-75dc0cb5641b:launchConfigurationName/vm-govtech-locus-stgez-nomads-20230905041931161400000001" -> (known after apply)
      ~ id                               = "vm-govtech-locus-stgez-nomads-20230905041931161400000001" -> (known after apply)
      ~ name                             = "vm-govtech-locus-stgez-nomads-20230905041931161400000001" -> (known after apply)
      - placement_tenancy                = "default" -> null # forces replacement
      - vpc_classic_link_security_groups = [] -> null
        # (10 unchanged attributes hidden)

      + ebs_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + no_device             = (known after apply)
          + snapshot_id           = (known after apply)
          + throughput            = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }

      + metadata_options { # forces replacement
          + http_endpoint               = "enabled" # forces replacement
          + http_put_response_hop_limit = 3 # forces replacement
          + http_tokens                 = "required" # forces replacement
        }

      ~ root_block_device {
          ~ encrypted             = false -> (known after apply)
          ~ iops                  = 0 -> (known after apply)
          ~ throughput            = 0 -> (known after apply)
            # (3 unchanged attributes hidden)
        }
    }

For vault cluster and consul cluster

Terraform will perform the following actions:

  # module.core.module.consul_servers.aws_autoscaling_group.autoscaling_group will be updated in-place
  ~ resource "aws_autoscaling_group" "autoscaling_group" {
      + force_delete_warm_pool    = false
        id                        = "vm-govtech-locus-stgez-consul20201008074307130700000016"
      ~ launch_configuration      = "vm-govtech-locus-stgez-consul-20230905083208680900000001" -> (known after apply)
        name                      = "vm-govtech-locus-stgez-consul20201008074307130700000016"
        tags                      = [
            {
                "key"                 = "Name"
                "propagate_at_launch" = "true"
                "value"               = "vm-govtech-locus-stgez-consul"
            },
            {
                "key"                 = "consul-servers"
                "propagate_at_launch" = "true"
                "value"               = "vm-govtech-locus-stgez-consul"
            },
        ]
        # (22 unchanged attributes hidden)
    }

  # module.core.module.consul_servers.aws_launch_configuration.launch_configuration must be replaced
+/- resource "aws_launch_configuration" "launch_configuration" {
      ~ arn                              = "arn:aws:autoscaling:ap-southeast-1:802909931189:launchConfiguration:ef4c6481-b72a-40f4-b1fa-31ce15f7696f:launchConfigurationName/vm-govtech-locus-stgez-consul-20230905083208680900000001" -> (known after apply)
      ~ id                               = "vm-govtech-locus-stgez-consul-20230905083208680900000001" -> (known after apply)
      ~ name                             = "vm-govtech-locus-stgez-consul-20230905083208680900000001" -> (known after apply)
      - vpc_classic_link_security_groups = [] -> null
        # (10 unchanged attributes hidden)

      + ebs_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + no_device             = (known after apply)
          + snapshot_id           = (known after apply)
          + throughput            = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }

      + metadata_options { # forces replacement
          + http_endpoint               = "enabled" # forces replacement
          + http_put_response_hop_limit = 3 # forces replacement
          + http_tokens                 = "required" # forces replacement
        }

      ~ root_block_device {
          ~ iops                  = 0 -> (known after apply)
          ~ throughput            = 0 -> (known after apply)
            # (4 unchanged attributes hidden)
        }
    }

  # module.core.module.vault.aws_autoscaling_group.autoscaling_group will be updated in-place
  ~ resource "aws_autoscaling_group" "autoscaling_group" {
      + force_delete_warm_pool    = false
        id                        = "vm-govtech-locus-stgez-vault20201008075653497200000002"
      ~ launch_configuration      = "vm-govtech-locus-stgez-vault-20230905041931186300000003" -> (known after apply)
        name                      = "vm-govtech-locus-stgez-vault20201008075653497200000002"
        # (22 unchanged attributes hidden)

        # (5 unchanged blocks hidden)
    }

  # module.core.module.vault.aws_launch_configuration.launch_configuration must be replaced
+/- resource "aws_launch_configuration" "launch_configuration" {
      ~ arn                              = "arn:aws:autoscaling:ap-southeast-1:802909931189:launchConfiguration:ebdf8f27-cbed-4a92-8887-54679c512c41:launchConfigurationName/vm-govtech-locus-stgez-vault-20230905041931186300000003" -> (known after apply)
      ~ id                               = "vm-govtech-locus-stgez-vault-20230905041931186300000003" -> (known after apply)
      ~ name                             = "vm-govtech-locus-stgez-vault-20230905041931186300000003" -> (known after apply)
      - vpc_classic_link_security_groups = [] -> null
        # (11 unchanged attributes hidden)

      + ebs_block_device {
          + delete_on_termination = (known after apply)
          + device_name           = (known after apply)
          + encrypted             = (known after apply)
          + iops                  = (known after apply)
          + no_device             = (known after apply)
          + snapshot_id           = (known after apply)
          + throughput            = (known after apply)
          + volume_size           = (known after apply)
          + volume_type           = (known after apply)
        }

      + metadata_options { # forces replacement
          + http_endpoint               = "enabled" # forces replacement
          + http_put_response_hop_limit = 3 # forces replacement
          + http_tokens                 = "required" # forces replacement
        }

      ~ root_block_device {
          ~ encrypted             = false -> (known after apply)
          ~ iops                  = 0 -> (known after apply)
          ~ throughput            = 0 -> (known after apply)
            # (3 unchanged attributes hidden)
        }
    }

Plan: 2 to add, 2 to change, 2 to destroy.

Tingweiftw · 2023-11-24T08:01:37Z

modules/nomad-cluster/main.tf

+  metadata_options {
+    http_endpoint               = "enabled"
+    http_tokens                 = "required"
+    http_put_response_hop_limit = 3


based on: https://support.hashicorp.com/hc/en-us/articles/1500005343862-Required-Additional-Configuration-When-Using-IMDSv2

When enabling IMDSv2 on a Terraform Enterprise EC2 host, it is required to set the http-put-response-hop-limit option key to a value of 2 or greater. This meta-data option key
limits the number of hops that metadata requests can travel across a network, and will affect Terraform Enterprise’s ability to interact with S3 buckets. This is required due to the Terraform Enterprise application being a set of containers running on the EC2 instance, making the minimum hop limit 2. Please note the exact setting required is entirely dependent on the environment that Terraform Enterprise is deployed in. The maximum value for the http-put-response-hop-limit key is 64, but testing may be required to determine an exact value to set the key to.

Tingweiftw · 2023-11-29T08:41:47Z

modules/core/nomad_servers.tf

@@ -9,8 +9,7 @@ locals {
 }

 module "nomad_servers" {
-  source  = "hashicorp/nomad/aws//modules/nomad-cluster"
-  version = "0.7.1"
+  source = "../nomad-cluster"


use the copied nomad-cluster terraform module which has the metadata_options. Spot price is set to an empty string which means maximum price of a the dedicated instance.

Tingweiftw · 2023-12-01T04:18:35Z

modules/consul-cluster/main.tf

+  spot_price    = var.spot_price
+
+  # added to https://github.com/hashicorp/terraform-aws-vault/tree/v0.14.1/modules/vault-cluster 
+  metadata_options {


addition to the copied source code to enable imdsv2

Tingweiftw · 2023-12-01T04:20:01Z

modules/vault-cluster/main.tf

+  user_data     = var.user_data
+
+  # added to https://github.com/hashicorp/terraform-aws-vault/tree/v0.14.1/modules/vault-cluster 
+  metadata_options {


addition to the copied source code to enable imdsv2

Tingweiftw · 2023-12-01T04:20:37Z

70d6b4d

removed from the vault-cluster, tags are not supported anymore

jrlonan-gt

LGTM, since this is mostly copied from a archived source code with the metadata option added

add metadata_options

9703c74

github-actions bot added feat size/XS labels Nov 24, 2023

Tingweiftw commented Nov 24, 2023

View reviewed changes

Tingweiftw closed this Nov 29, 2023

Tingweiftw reopened this Nov 29, 2023

Tingweiftw added 2 commits November 29, 2023 16:13

nomad server to use copied nomad-cluster

dc73fdc

add spot price

d17fb04

github-actions bot added size/S and removed size/XS labels Nov 29, 2023

empty string for dedicated instance price

e74e8bc

Tingweiftw commented Nov 30, 2023

View reviewed changes

copy vault cluster source code

f9212df

github-actions bot added size/XXL and removed size/S labels Dec 1, 2023

Tingweiftw added 6 commits December 1, 2023 10:23

inout for vault-cluster

43eb71f

add in vault-security-group

bf4fe37

add in copied consul-cluster source code

d41d938

fmt

152a19a

add consul-client-sg-rules

4c69524

remove unsupported tags

70d6b4d

Tingweiftw commented Dec 1, 2023

View reviewed changes

jrlonan-gt approved these changes Dec 1, 2023

View reviewed changes

Tingweiftw merged commit 91448eb into master Dec 1, 2023
3 checks passed

Tingweiftw deleted the feat--nomad-enable-imdsv2 branch December 1, 2023 09:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(nomad): add metadata_options to enable imdsv2 #297

feat(nomad): add metadata_options to enable imdsv2 #297

Tingweiftw commented Nov 24, 2023 •

edited

Loading

Tingweiftw Nov 24, 2023

Tingweiftw Nov 29, 2023

Tingweiftw Dec 1, 2023

Tingweiftw Dec 1, 2023

Tingweiftw commented Dec 1, 2023

jrlonan-gt left a comment

feat(nomad): add metadata_options to enable imdsv2 #297

feat(nomad): add metadata_options to enable imdsv2 #297

Conversation

Tingweiftw commented Nov 24, 2023 • edited Loading

Motivation and Context

Description

How has this been tested?

For nomad servers (don't have the diff for nomad client, but similar)

For vault cluster and consul cluster

Tingweiftw Nov 24, 2023

Choose a reason for hiding this comment

Tingweiftw Nov 29, 2023

Choose a reason for hiding this comment

Tingweiftw Dec 1, 2023

Choose a reason for hiding this comment

Tingweiftw Dec 1, 2023

Choose a reason for hiding this comment

Tingweiftw commented Dec 1, 2023

jrlonan-gt left a comment

Choose a reason for hiding this comment

Tingweiftw commented Nov 24, 2023 •

edited

Loading